Skip to content

ci: seed test-e2e's warm DerivedData from main every 6 hours - #14082

Merged
teamleaderleo merged 3 commits into
mainfrom
ci/e2e-test-only-product-reuse
Sep 24, 2026
Merged

teamleaderleo merged 3 commits into
mainfrom
ci/e2e-test-only-product-reuse

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

A focused test-e2e.yml dispatch should compile only what changed. #14016 made that possible: a build starts from DerivedData that a main-contained dispatch published. But nothing publishes that DerivedData regularly. Focused dispatches test pull request revisions, which never publish, and the seeds so far came from hand dispatches, whose artifacts expire after 3 days. When no seed exists, the build logs "no-main-derived-data" and compiles the whole app.

Changes

  • e2e-derived-data-seed.yml (new). Every 6 hours it dispatches test-e2e.yml on main's HEAD with cmuxTests/CmuxPopoverGroupTests (36 ms), unless that SHA already has a seed run that finished or is still running. It dispatches rather than calls, so the run has the identity e2e_warm_derived_data.py trusts: path test-e2e.yml, event workflow_dispatch, branch main. This mirrors how ci-main-full-suite.yml dispatches ci.yml.
  • Seed runs always compile. Only a compile publishes DerivedData. A seed run could adopt an exact product instead, because product identity ignores .github/ and tests/, and a focused dispatch of the pre-merge PR head is a valid producer. It would then publish nothing. test-e2e.yml now skips product reuse when dispatch_id starts with e2e-derived-data-seed-.
  • Not in this PR: clearing a half-adopted DerivedData after the adopt step times out. Main landed that while this was open, and the merge takes main's version.

Validation

  • All 133 commands in ci-guards.yml run locally, including those inside run: | blocks. bun test/claude-environment.test.ts fails, as it does on main. test_local_build_cache_preflight.py failed once under sweep load and passes 22 of 22 when run alone; its tests are deadline-based.
  • actionlint is clean.
  • An independent review found the same warm key for blacksmith-6vcpu-macos-26 and the 12 vCPU pool that ci: route focused tests to one front door, and half of them to 12 vCPU #14067 routes odd SHAs to. Both run image 26-20260916 with Xcode 26.6 and /Users/runner/_work/cmux/cmux, and both computed f9243ea2… (runs 35939305209 and 35939623231). An image rollout that reaches one pool first would split the key until the other catches up.
  • The first seed was dispatched by hand: run 35941008193, on main a3b7014. Its dispatch id isn't a seed id, so the first scheduled tick may run one more build.

Not yet shown

  • ci(e2e): start builds from main's DerivedData so test-only changes skip the app compile #14016's open question: whether Xcode 26 rebuilds only the changed test file from adopted DerivedData. The first focused dispatch after a seed answers it; the build summary reports the adoption and how many inputs differed.
  • Changing test-e2e.yml changes the E2E product recipe fingerprint, so exact-product reuse misses once after merge.
  • Cost: one e2e build every 6 hours on free Blacksmith, skipped when main hasn't moved.

— Camera g1 🛠️

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • End-to-end testing now includes scheduled runs that prepare derived data, alongside the existing manual test runs. These scheduled runs use the latest code on the main branch and avoid reusing compiled products for the seed run. No changes to app functionality are included.

#14016 lets a test-e2e.yml build start from DerivedData that a
main-contained dispatch published, so a test-only change recompiles only
the tests. Only such a dispatch publishes it, and focused dispatches test
pull request revisions, so none ever did: builds report
"no-main-derived-data" and compile the whole app from empty DerivedData
(e.g. 12.4 min of build for a 36 ms suite in run 35939043846).

Dispatch test-e2e.yml on main's HEAD with one fast suite every 6 hours,
skipping a SHA that already has a seed dispatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 24, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 1c8e9400-381d-403c-98c2-ac5842a4e42e

📥 Commits

Reviewing files that changed from the base of the PR and between 9dc50a9 and d2fdcce.

📒 Files selected for processing (2)
  • .github/workflows/e2e-derived-data-seed.yml
  • .github/workflows/test-e2e.yml
 _________________________________
< Pvt. Rabbit reporting for duty! >
 ---------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

A seed dispatch could adopt an exact product, skip the compile, and so
publish nothing; seed dispatches now skip product reuse. A step timeout in
"Adopt main's DerivedData" kills the restore before its own cleanup, so the
compile could start on a half-extracted tree; a failed adoption now clears
the DerivedData first. The seed dedupe now looks back 300 runs (50 covered
about two hours) and retries only runs that never finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo enabled auto-merge (squash) September 24, 2026 01:22
…uct-reuse

# Conflicts:
#	.github/workflows/test-e2e.yml
@teamleaderleo
teamleaderleo merged commit 19144b6 into main Sep 24, 2026
47 of 48 checks passed
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Finding from #14098 that affects this PR: warm DerivedData adoption only saves compile time when the producer commit and the tested commit have the same app and package sources.

Tonight, the 3 test-e2e builds that adopted main's DerivedData (runs 35942257134, 35942449623, 35942433848) still compiled 2,510 app-target files, which is what a cold build compiles. They also spent about 2 minutes downloading first. Their producer was main about 9 hours earlier, with about 250 changed inputs in between. When a package's interface changes, the Swift build recompiles every file that imports that package, and that is nearly every app file.

Adoption paid off only once (6 min instead of 22), and in that case the producer was a few commits away.

So a seed that is refreshed every 6 hours will rarely produce a hit on PR revisions, because many PRs merge into main between seeds. #14098 now checks the diff before downloading and starts cold when app sources differ.

teamleaderleo added a commit that referenced this pull request Sep 24, 2026
e2e-derived-data-seed.yml (#14082) landed with an ungated Blacksmith
fallback; the fork-fallback guard caught it on rebase. Also address review:
docs/ci-runners.md still showed the ungated runs-on example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant